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I. INTRODUCTION 



The School Breakfast Program (SBP), authorized by the Child Nutrition Act of 1966, started 
as a pilot program to provide funding for school breakfasts in poor areas and areas where children 
had to travel great distances to school. The intent was to provide a nutritious breakfast to children 
who might otherwise not receive one. The importance of a nutritious breakfast is supported by 
several studies that have linked breakfast to improved dietary status and enhanced school 
performance. More recent research suggests that providing school breakfasts to low-income children 
is associated with the greater likelihood of eating a substantive breakfast, improved school 
attendance, and decreased tardiness (Pollitt and Mathews 1998; and Briefel et al. 1999). Less is 
known about the effects of school breakfasts on children’s cognitive functioning and academic 
achievement, although some studies suggest that school breakfasts may lead to improvements in 
these outcomes (Briefel et al. 1999). 

In response to the body of evidence suggesting that school breakfasts provide dietary and 
educational benefits, many observers have urged that school breakfasts be, more widely available. 
Within this context, Congress passed Section 109 of the William F. Goodling Child Nutrition Act 
of 1998 (P.L. 105-336). This authorizes the Secretary of Agriculture, through the Food and 
Nutrition Service (FNS), U.S. Department of Agriculture (USDA), to conduct a demonstration and 
evaluation that will rigorously assess the effects of the Universal-Free School Breakfast Program 
(USBP) on program participation and a broad range of student outcomes, including academic 
achievement, school attendance and tardiness, classroom behavior and attentiveness, and dietary 
status. 





L 



A. THE USBP DEMONSTRATION AND EVALUATION 



The Child Nutrition Act of 1998 authorized demonstration pilot projects in up to six school food 
authorities (SFAs) and a rigorous evaluation to assess the effects of providing free school breakfasts 
to elementary school children. The evaluation is to address four main research objectives: 



1 . Determine the effect that participation by elementary schools in the USBP pilot projects 
has on student participation in the breakfast program, on the costs to the federal 
government in increases in funding for reimbursing program meals, and on simplifying 
paperwork the schools must complete and other administrative procedural participation 
requirements 

2. Assess the effects of the USBP in elementary schools on selected student outcomes, 
including academic achievement, school attendance and tardiness, classroom behavior 
and discipline, and dietary intakes over the course of a day 

3. Compare the characteristics and outcomes of USBP participants with those of 
nonparticipating children 

4. Compare the characteristics and outcomes of USBP participants with children 
participating in the regular SBP 



FNS has contracted with Mathematica Policy Research, Inc. to develop a comprehensive, 
rigorous study design for evaluating the USBP pilot programs. This report describes the evaluation 
strategy, which includes a methodologically rigorous design, a well-defined set of outcome variables, 
and data collection plans comprehensive enough to allow the gathering of evidence on program 
effectiveness. 

The evaluation strategy is a randomized design: it first pairs schools, then randomly assigns 
each of the schools in each pair to either a treatment (USBP) or control (regular SBP) group. Next, 
samples of students are drawn from each school. The evaluation plan involves a wide range of data 
collection activities, including the collection of school administrative data, surveys of students and 
their parents, extraction of school records data, administration of cognitive tests and student 



achievement tests, and site visits. Because the actual school districts applying for the demonstration 
are unknown at this time, the design considers two possible sample sizes for the number of schools. 
The first version assumes student achievement data will be available at two or more points in time 
for sampled students from tests routinely administered by the school districts chosen for the study. 
The second version would use district-administered achievement tests to supply baseline data and 
would obtain follow-up data on student achievement through tests administered by the evaluation 
contractor. In addition, because it is uncertain whether the demonstration will start in September 
2000 or in January 2001, the design includes an option to conduct a preimplementation survey of 
students’ parents. This survey would only be conducted if demonstration implementation is delayed 
until January 2001. 

B. CHALLENGES TO THE USBP EVALUATION 

Several challenges to the USBP evaluation must be addressed if the design is to yield valid and 
policy-relevant evidence on the effects of providing free school breakfasts to elementary school 
children. One challenge warrants special attention: whether, given the resources available for the 
demonstration and evaluation, the preferred design has the statistical precision to estimate impacts 
on student outcomes, especially student achievement. Because the USBP is generally considered 
an intervention at the school level, the preferred approach to evaluating the program is to use schools 
as the unit of assignment. Even under the best available designs, however, the probability of being 
able to detect policy-relevant effects is limited by how many schools the evaluation resources can 
serve. 
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Precision levels obtainable in the experimental-based evaluation strategy using schools as the 
unit of assignment are limited for two reasons: (1) the majority of students do not change breakfast 
program participation status (the “dilution” effect), and (2) variation exists between schools. 

1. The “Dilution” Effect 

The majority of students at the participating schools are unlikely to start receiving school 
breakfasts under the USBP. Many of them— on average 25 or 30 percent— will already be 
participating in the regular SBP, and it is expected that they will continue to receive breakfast under 
the USBP. The introduction of free school breakfasts is not likely to substantially affect their 
outcomes. Among the other students, a significant number will choose not to start eating a school 
breakfast, even if it is now free, because their families will continue to provide breakfast at home or 
for other reasons. The precision calculations are based on an assumption that the increase in school 
breakfast participation will be approximately 25 percentage points (from 30 to 55 percent). In the 
experimental versus control comparisons, which are the essence of the full application of the 
experimental design, the effects of school breakfast on those 25 percent that are new participants are 
diluted by the 75 percent of students whose participation is not changed. 1 

The design report outlines two possible approaches to dealing with this problem. The first is 
to conduct a preimplementation survey at the schools to identify the children most likely to 
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'Based on the information available, the 25 percent increase in participation represents a 
reasonable assumption. However, if FNS or others can supply information supporting a higher 
assumption, this would lower the estimates of minimum detectable effects. The effects are 
proportional— with a 50 percentage point rather than a 25 percentage point increase in participation, 
minimum effect sizes would be divided by 2. However, given that the program is likely to attract 
additional participants who previously did not choose to eat a school breakfast, many of whom may 
already be eating a nutritious breakfast at home, it is possible that program impacts may be smaller 
with large increases in participation. If so, even though the statistical precision of the evaluation 
design increases with larger increases in participation, the policy-relevant program impacts may be 
smaller than those reported currently in the literature. 
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participate and focus the analysis on those students. The second is to use analysis approaches that 
are not based directly on the experimental design. In particular, such nonexperimental methods 
would take advantage in the analysis of the fact that one can observe which students actually 
participate in the USBP, thus making it possible to sharpen the comparisons in the analysis (though 
at the cost of losing some advantages of the pure experimental design ). 2 

2. Variation Between Schools 

The other main problem that limits the precision in the currently planned demonstration is that, 
even with pairing of schools prior to randomization, there are likely to be significant differences 
between the schools. This variation tends to confound the analysis, because it is correlated with 
treatment status. For instance, if the treatment school in one of the school pairs happens to have 
children with relatively high nutrient intakes, the treatment group as a whole will reflect this, making 
it difficult to disentangle school differences from the effects of the USBP. 

For a given number of schools, adding more children per school to the design has only limited 
effects in solving this problem. Even if data were available on all the children in a school and, 
hence, the evaluation could measure the school means perfectly, there would still only be a limited 
number of school observations with which to estimate treatment effects. 



2 Another possible approach to dealing with the dilution problem would be to require all SFAs 
participating in the demonstration to essentially enforce student participation by offering an intensive 
treatment in which all students would receive a breakfast in their classrooms, with no options for not 
participating. After careful consideration of this approach, this option is not recommended because 
of two significant research risks associated with it. First, it is unlikely that most of the SFAs 
applying for the program would agree to operate this variant of the USBP. Second, even if they did 
agree, many of the students forced to participate might throw their breakfasts away, thus, the 
intended effects of the intensive treatment and possibly leading to incorrect analysis inferences. 
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The evaluation design presented in this report addresses this main challenge to the evaluation, 
as well as the many other measurement and data collection challenges inherent to an evaluation of 
this magnitude. 

C. OVERVIEW OF THE REPORT 

The rest of the report is organized in four parts and provides a detailed strategy for evaluating 
the USBP pilot projects. The rest of Part 1 provides background information, describes the SBP, 
summarizes the literature on breakfast and learning, and discusses the rationale for the USBP. It also 
identifies the requirements of the demonstration and evaluation, provides a conceptual framework 
of how participation in a USBP may affect student outcomes, and provides an overview of the full 
set of evaluation design components. 

Part 2 describes the design for the impact study. First, it identifies research questions to be 
addressed and the hypotheses to be tested; then, it discusses outcomes and the measures that will be 
used to assess program effectiveness. It describes the approach for evaluating the USBP 
demonstration programs, discusses evaluation and sampling design issues and strategies, presents 
an analysis plan for assessing program impacts, and describes data collection plans. 

Part 3 presents the approach for conducting the implementation study of the USBP pilot 
projects. It describes study topics and research questions and outlines plans for process data 
collection. Also described are analysis strategies for assessing program implementation and whether 
the goals of administrative simplification are being met. 

Part 4 provides an assessment of the feasibility of implementing the preferred evaluation design. 
It reviews the statistical precision achieved, provides an estimate of the demonstration costs, and 
identifies strengths and weaknesses, as well as risks. 
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II. BACKGROUND 



In order to develop a design that results in an evaluation fully responsive to the needs of the 
Secretary of Agriculture and Congress, it is important to be clear about the demonstration and 
evaluation requirements specified in the legislation authorizing the Universal-Free School Breakfast 
Program (USBP) pilot projects. This chapter provides background information for the USBP 
demonstration and evaluation. Section A describes the School Breakfast Program (SBP), while 
Section B summarizes the literature on breakfast consumption and student learning. Section C 
provides the rationale for the USBP demonstration. Section D identifies and discusses 
demonstration and evaluation requirements. Section E describes a conceptual framework of how 
a universal-free school breakfast program would affect students and schools. Finally, Section F 
provides an overview of the evaluation components. 

A. OVERVIEW OF THE SCHOOL BREAKFAST PROGRAM 

The SBP is a federal program that provides states with cash assistance (and commodities) for 
breakfast programs in schools. It is administered at the federal level by the Food and Nutrition 
Service (FNS). State education agencies and local school food authorities (SFAs) administer the 
program at the local level. Originally, it was a pilot program that targeted children from low-income 
school districts. With the 1975 amendments to the Child Nutrition Act of 1966, the SBP became 
permanent. The objective of Congress was to make the program available in all schools where it was 
needed, to provide adequate nutrition for children in attendance. Congress passed the 1989 Child 
Nutrition Act to expand the availability of the SBP, requiring the Secretary of Agriculture to award 
start-up funding (ranging from $3 to $5 million per school district) to states wishing to implement 




